Skip to content

fix(p2p): disconnect peers whose gossipsub send queue never drains - #739

Merged
pablodeymo merged 14 commits into
mainfrom
fix/p2p-disconnect-slow-gossipsub-peers
Nov 18, 2026
Merged

pablodeymo merged 14 commits into
mainfrom
fix/p2p-disconnect-slow-gossipsub-peers

Conversation

@MegaRedHand

@MegaRedHand MegaRedHand commented Oct 30, 2026 •

Copy link
Copy Markdown
Collaborator

🗒️ Description / Motivation

A heap profile of our mainnet follower (2.8 GB steady RSS) showed about 490 MB of gossipsub message copies that never went away: they sat in the send queues of a few peers whose gossipsub connection handler had disabled itself.

The mechanism, in the pinned rust-libp2p (same in upstream master):

  • The handler counts every inbound and outbound substream event, successful ones included, and after MAX_SUBSTREAM_ATTEMPTS sets itself to Disabled(MaxSubstreamAttempts). In that state poll is Pending forever, so the send queue is never read again.
  • The behaviour is not told. It keeps queueing forwards for the peer until the per-peer queue is full.
  • A queued message is only dropped when the handler takes it off the queue, so the copies stay in memory until the connection closes. Other protocols keep using that connection, so nothing closes it.

Gossipsub does report such a peer: Event::SlowPeer every heartbeat. We dropped that event. Gossipsub's own score penalty for it (slow_peer_weight) left the peer in the negative band, nowhere near the graylist. Lighthouse handles the event in its peer manager instead.

What Changed

  • crates/net/p2p/src/swarm_adapter.rs:
    • SlowPeers collects SlowPeer reports between the existing 10 s checks. A report counts only when its priority or its non-priority failures exceed SLOW_PEER_REPORT_THRESHOLD (10), the cut Lighthouse applies before it penalizes a slow peer.
    • disconnect_slow_peers runs on that tick, right after disconnect_graylisted_peers. It disconnects a peer reported in each of the last SLOW_PEER_CHECK_LIMIT (6) checks, so about a minute without a break.
    • One check disconnects at most MAX_SLOW_PEER_DISCONNECTS_PER_CHECK (10) peers, the ones with the most failed messages first. The others keep their streak, so a peer still failing at the next check is disconnected then. Peers held back by the cap are logged as a warning.
    • The event is still forwarded to the P2P actor as before.
  • crates/net/p2p/src/metrics.rs: adds the lean_gossipsub_slow_peer_disconnects_total counter.
  • docs/beacon_wire.md, docs/metrics.md, CLAUDE.md: describe the rule, its two limits and the metric.

Correctness / Behavior Guarantees

  • Beacon only. The gate is the one the graylist check already uses: peer scoring on. Lean is unchanged.
  • Only a persistent failure disconnects. A check with no counted report resets the peer's streak, so a peer that is briefly slow keeps its connection.
  • Our own slowness does not drop the whole peer set (review comment). Gossipsub counts a forward that expired in a peer's queue (connection_handler_forward_duration) as the same non_priority failure as a full queue, so a saturated uplink or a stalled runtime on our side makes every peer look slow at once. The threshold filters short stalls, and the cap bounds how many peers one check can drop. A sustained slowdown can still cost up to MAX_SLOW_PEER_DISCONNECTS_PER_CHECK peers per check while discovery refills them. We deliberately leave that case alone: bandwidth so slow that it drains our peers is not something we can solve from our side.
  • Known gaps:
    • Some peers keep their pinned queue until the connection closes, as today. These are peers whose handler is dead but whose failures stop within the minute, for example once they are pruned from the mesh and score below the gossip threshold.
    • A dead peer that gets few messages per heartbeat, for example one gossipsub has already pruned from the mesh, may stay under SLOW_PEER_REPORT_THRESHOLD and keep its connection.
  • Since fix: cap gossipsub's log output at ERROR unless RUST_LOG raises it #719 caps libp2p_gossipsub logs at ERROR, gossipsub's "Send Queue full" WARNs no longer show by default. An operator now sees these peers through this change's info! disconnect line and the new counter.

Tests Added / Run

  • Eight unit tests on SlowPeers:
    • a peer failing in every check is returned at the limit;
    • a quiet check resets the streak;
    • several reports in one check count once;
    • a returned peer starts a new streak;
    • a report at the threshold does not count, and resets the streak;
    • the threshold applies to each count on its own (priority alone is enough; the two added up are not);
    • one check returns at most the cap, most failed first, and a deferred peer that keeps failing is returned at the next check;
    • a deferred peer that stops failing loses its streak.
  • Replayed the follower's log in 10 s bins (before the threshold and cap were added):
    • The three stuck peers had unbroken runs of "Send Queue full" of 29, 54 and 377 bins (5 to 63 minutes), so each would have been disconnected within about a minute.
    • A peer with 3 isolated failures would have kept its connection.
    • The bins do not show whether those peers had more than SLOW_PEER_REPORT_THRESHOLD failures within a single 700 ms heartbeat, so the replay does not yet confirm the threshold still catches them.
  • cargo test --profile release-fast -p ethlambda-p2p --lib: 370 passed, 1 ignored.

Related Issues / PRs

  • One of three memory fixes from the same profile; the other two are separate PRs (vote-table growth, state encoding).

✅ Verification Checklist

  • Ran make fmt — clean
  • Ran make lint (clippy with -D warnings) — clean. Ran cargo clippy --locked -p ethlambda-p2p --all-targets -- -D warnings (clean), not the full workspace target
  • Ran make test (test-consensus plus test-node, at release-fast) — all passing. Ran only the p2p lib tests above

A gossipsub connection handler disables itself for good after a few
substream negotiations on one connection, successful ones included, and
then never takes another message off its send queue. The behaviour is
not told, so it keeps queueing forwards for that peer until the queue is
full, and a queued message is only dropped when it is taken off. The
copies stay in memory until the connection closes, which other protocols
can keep open for hours: on the mainnet follower these queues held about
490 MB.

Gossipsub reports such a peer as SlowPeer every heartbeat, and the node
ignored the event. The swarm task now collects those reports and, on the
10 s check that already drops graylisted peers, disconnects a peer that
was reported in each of the last six checks. The follower's three stuck
peers failed without a break for 5 to 63 minutes, so each would have
been dropped within about a minute, while a peer with a few isolated
failures keeps its connection.

Beacon only, gated like the graylist check on peer scoring being on.
@github-actions

Copy link
Copy Markdown

🤖 Claude Code Review

Review: PR 739, disconnect peers whose gossipsub send queue never drains

The change is small, self-contained and well reasoned. I found no blocking issues. I read the diff only and did not build it or run the tests.

Correctness

  • SlowPeers::check is sound. It drains reported on every call, so that map can't grow. streaks.retain drops any peer not reported since the last check, which gives the "consecutive" semantics. Peers that disconnect on their own are cleaned up on the next tick.
  • Several SlowPeer events inside one 10 s window count as one check. This is covered by the test several_reports_within_one_check_count_once_towards_the_streak.
  • The streak is removed once a peer is returned. If disconnect_peer_id fails because the peer is already gone, nothing leaks.
  • swarm.disconnect_peer_id closes every connection to the peer. That is the intended way to free the handler's queue, and a reconnect gets a fresh handler.
  • The event is recorded in swarm_loop before being forwarded to the actor, so the actor still sees SlowPeer. The tracking has no dependency on actor behaviour.

Points worth a look (non-blocking)

  1. Beacon-only gating (swarm_adapter.rs, disconnect_slow_peers): peer_score(local_peer_id).is_none() is used as a stand-in for "scoring is off, so this is lean". This presumably mirrors disconnect_graylisted_peers, and the doc comment says so. It is indirect, though. If gossipsub ever returns Some for the local id, or None for an unrelated reason, the gate flips silently. A small shared helper such as scoring_enabled(&swarm) used by both functions would make the coupling explicit.
  2. False positives on a healthy-but-loaded peer: the doc notes that the SlowPeer report doesn't distinguish a refused message from an expired one. Six consecutive 10 s checks, about a minute, is a reasonable guard. A peer on a congested link during a long mainnet burst could still hit it. This is a trade-off I'd accept, since the lean_gossipsub_slow_peer_disconnects_total metric will show if it fires too often. Making the limit a named constant, as the PR already does, makes tuning easy.
  3. Tick timing: with MissedTickBehavior::Skip, a stalled loop can make "consecutive checks" cover more than 10 s each. This only makes disconnects slightly slower, so it is harmless.
  4. Log level: info! per disconnect is fine at expected rates. If a bad actor can trigger it repeatedly by reconnecting, consider debug! or rate-limiting. The metric already covers that case.

Tests and docs

  • The four unit tests cover the limit, a reset on a gap, the same-window collapse, and a restart after a return. That is the full state machine.
  • docs/beacon_wire.md, docs/metrics.md and CLAUDE.md are updated consistently, and the metric name follows the lean_ convention.
  • The FailedMessages sum priority + non_priority is a plain usize add. It can't realistically overflow, and the value is only used for logging.

Looks good to merge. Point 1 is the only change I'd suggest making.


Automated review by Claude (Anthropic) · sonnet · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Codex Code Review

Looks good overall. The change is narrowly scoped to P2P hygiene, not consensus logic, and the approach is reasonable for mitigating the documented gossipsub queue retention issue.

  • No consensus-layer concerns found: nothing here touches fork choice, attestation validation, state transition, finalization, XMSS, or SSZ.
  • Security/perf direction looks good: disconnecting persistently SlowPeer peers should bound retained queued-message memory and reduce a plausible resource-exhaustion vector.

One thing I’d flag:

  • crates/net/p2p/src/swarm_adapter.rs:414 — disconnect_peer_id(peer_id).is_ok() only increments the metric/logs on success, but a failed disconnect leaves the peer out of slow_peers.streaks because check() forgets persistent peers before the disconnect attempt. If disconnect_peer_id fails transiently, that peer gets a full reset and may retain the pinned queue for another minute before being retried. Consider keeping the peer in the tracked set until the disconnect succeeds, or re-inserting it on failure.

Nice-to-have:

  • crates/net/p2p/src/swarm_adapter.rs:370 — priority + non_priority is fine in practice, but saturating_add would be a slightly more defensive choice for bookkeeping code.
  • crates/net/p2p/src/swarm_adapter.rs:637 — the test named “count once towards the streak” only checks that many reports in one interval do not immediately trip the threshold; it does not verify the streak increments by exactly one across successive checks. A follow-up assertion over two intervals would make the behavior more explicit.

If you want, I can turn the first point into a suggested patch.


Automated review by OpenAI Codex · gpt-5.4 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Kimi Code Review

I'll review this PR which adds slow peer disconnection for gossipsub peers with dead streams that accumulate queued messages in memory.

Overall Assessment

This is a well-structured, targeted fix for a memory leak caused by gossipsub connection handlers that disable themselves but leave queues pinned in memory. The logic is sound, tests are comprehensive, and documentation is thorough. I have a few minor concerns.


Issues Found

1. Race condition in SlowPeers::check() — stale FailedMessages aggregation (swarm_adapter.rs:372-380)

fn check(&mut self) -> Vec<(PeerId, usize)> {
    let reported = std::mem::take(&mut self.reported);
    self.streaks
        .retain(|peer_id, _| reported.contains_key(peer_id));  // line 374

Problem: The retain on line 374 uses reported.contains_key(peer_id), but reported was just taken. This is correct for removing peers not in the new report. However, there's a subtle issue: if a peer was in streaks with (checks=5, total=10), and is reported again with failed=0 (possible? FailedMessages sums to 0?), the streak continues with no new failures. But record() uses += failed, so failed=0 would still create a reported entry via or_default().

Actually, re-reading: failed_messages.priority + failed_messages.non_priority could be 0 if both are 0. The or_default() would make reported[peer] = 0, and contains_key would be true. This seems harmless but slightly wasteful.

More serious: The retain happens before updating streaks. Consider: peer A has streak (5, 10), not reported this check. retain removes it. Correct. But what if between take and retain, we cared about consistency? We don't—this is single-threaded.

Actual concern: The failed count returned includes only messages from checks where the peer was consecutively reported. If a peer fails check 1, misses check 2, fails checks 3-8, the streak resets. This is documented behavior, but the total in the returned tuple only sums the successful streak. The log message says "failed" without clarifying it's only the streak's failed count, not lifetime failures. Minor.

2. Missing peer_score check logic inconsistency (swarm_adapter.rs:402-408)

if swarm
    .behaviour()
    .gossipsub
    .peer_score(swarm.local_peer_id())
    .is_none()
{
    return;
}

Problem: This checks if our own peer has a score, not if gossipsub scoring is enabled for the peer being disconnected. The comment says "Beacon only, like disconnect_graylisted_peers", but disconnect_graylisted_peers doesn't have this check—it checks peer_score(peer_id) for each peer to get the score.

Wait, let me re-read disconnect_graylisted_peers... Actually it does:

let Some(score) = swarm.behaviour().gossipsub.peer_score(&peer_id) else {
    continue;
};

So disconnect_graylisted_peers checks per-peer score existence. The slow peer check uses peer_score(swarm.local_peer_id()) which returns None if scoring isn't enabled at all (e.g., lean mode). This is a proxy for "is this beacon mode?"

This is correct but fragile. If gossipsub ever returns a score for local_peer_id() in lean mode, this breaks. A more explicit mode check would be clearer. However, this matches the existing pattern elsewhere, so acceptable.

3. Test a_peer_failing_in_every_check_is_returned_at_the_limit off-by-one risk (swarm_adapter.rs:622-631)

for _ in 1..SLOW_PEER_CHECK_LIMIT {  // runs 5 times for limit=6
    slow_peers.record(peer, &failed(2));
    assert!(slow_peers.check().is_empty());
}
slow_peers.record(peer, &failed(2));
let expected = 2 * usize::try_from(SLOW_PEER_CHECK_LIMIT).unwrap();
assert_eq!(slow_peers.check(), vec![(peer, expected)]);

Problem: The loop runs SLOW_PEER_CHECK_LIMIT - 1 times (1..6 = 5 iterations for limit=6). Then one more record + check triggers at limit. Total records: 6, checks: 6. The streak reaches 6, and *checks >= SLOW_PEER_CHECK_LIMIT (6 >= 6) triggers.

But wait—the first check() returns empty, and checks becomes 1. After 5 loop iterations, checks is 5. The 6th record + check makes it 6, triggers. This is correct.

However, the test name says "at the limit" but the assertion checks >= limit. The test verifies exactly at limit. Good.

4. Test a_returned_peer_starts_a_new_streak has suspicious loop (swarm_adapter.rs:668-677)

for _ in 0..SLOW_PEER_CHECK_LIMIT {
    slow_peers.record(peer, &failed(1));
    slow_peers.check();
}
slow_peers.record(peer, &failed(1));
assert!(slow_peers.check().is_empty());

Problem: Each iteration: record then check. On iteration 6 (0-indexed, SLOW_PEER_CHECK_LIMIT=6), record makes checks=6, check returns the peer and removes it. So after the loop, the peer is gone from streaks. Then record + check starts fresh at 1, returns empty. Correct.

But this test doesn't verify the peer was actually returned in the loop! It should assert something about the loop iterations. As written, if check() never returned anything (bug), the test would still pass because the final state is empty. This test has a hole.

Suggested fix:

#[test]
fn a_returned_peer_starts_a_new_streak() {
    let peer = PeerId::random();
    let mut slow_peers = SlowPeers::default();
    for i in 0..SLOW_PEER_CHECK_LIMIT {
        slow_peers.record(peer, &failed(1));
        let result = slow_peers.check();
        if i < SLOW_PEER_CHECK_LIMIT - 1 {
            assert!(result.is_empty(), "should not trigger before limit");
        } else {
            assert_eq!(result, vec![(peer, SLOW_PEER_CHECK_LIMIT as usize)]);
        }
    }
    slow_peers.record(peer, &failed(1));
    assert!(slow_peers.check().is_empty(), "new streak should not trigger immediately");
}

5. Memory growth in SlowPeers::reported between checks (swarm_adapter.rs:357)

fn record(&mut self, peer_id: PeerId, failed_messages: &FailedMessages) {
    let failed = failed_messages.priority + failed_messages.non_priority;
    *self.reported.entry(peer_id).or_default() += failed;
}

Problem: If gossipsub emits SlowPeer for many unique peers between checks, reported grows unbounded. The check() clears it, but if the event loop is busy and check is delayed, this could accumulate. However, SlowPeer is emitted per heartbeat (700ms), and checks are every 10s, so max ~14 heartbeats of reports. With mesh size 8, this is bounded. Acceptable.

6. No upper bound on failed count (swarm_adapter.rs:358)

failed_messages.priority + failed_messages.non_priority as usize could theoretically overflow if gossipsub reports huge numbers. In practice, queue capacities are small (thousands), so this won't happen. Not a real concern.


Positive Observations

  1. Excellent documentation — The SlowPeers struct doc comment (lines 339-358) clearly explains the gossipsub handler bug, memory impact ("490 MB"), and design rationale.

  2. Conservative threshold — SLOW_PEER_CHECK_LIMIT = 6 (60 seconds) avoids disconnecting catch-up peers with transient queue pressure.

  3. Proper metrics and logging — Counter incremented only on successful disconnect, with structured logging.

  4. Test coverage — Four tests cover: exact limit triggering, streak reset, coalescing within one check, and fresh start after disconnect.

  5. Lean mode safety — The peer_score check prevents disconnections on lean chain, matching existing pattern.


Summary

Item Severity File:Lines Description
1 Low swarm_adapter.rs:668-677 Test a_returned_peer_starts_a_new_streak doesn't verify peer was actually returned during loop
2 Info swarm_adapter.rs:402-408 peer_score(local_peer_id) mode check is correct but implicit; consider explicit mode enum
3 Info swarm_adapter.rs:339-358 Doc comment mentions "490 MB" — consider if this is mainnet-specific and may need updating

Recommendation: Fix test in Item 1 to assert on loop behavior. Otherwise, this PR is ready to merge.


Automated review by Kimi (Moonshot AI) · kimi-k2.6 · custom prompt

MegaRedHand added a commit that referenced this pull request Oct 30, 2026
)

A gossipsub connection handler disables itself for good after a few
substream negotiations on one connection, successful ones included, and
then never takes another message off its send queue. The behaviour is
not told, so it keeps queueing forwards for that peer until the queue is
full, and a queued message is only dropped when it is taken off. The
copies stay in memory until the connection closes, which other protocols
can keep open for hours: on the mainnet follower these queues held about
490 MB.

Gossipsub reports such a peer as SlowPeer every heartbeat, and the node
ignored the event. The swarm task now collects those reports and, on the
10 s check that already drops graylisted peers, disconnects a peer that
was reported in each of the last six checks. The follower's three stuck
peers failed without a break for 5 to 63 minutes, so each would have
been dropped within about a minute, while a peer with a few isolated
failures keeps its connection.

Beacon only, gated like the graylist check on peer scoring being on.

Cherry-picked from fix/p2p-disconnect-slow-gossipsub-peers at 380fd29.
@pablodeymo

Copy link
Copy Markdown
Collaborator

One availability risk worth weighing before merging.

In the pinned fork, a forward that sits in the handler's queue longer than
connection_handler_forward_duration (1 s) is reported as non_priority:
handler.rs hits the timeout and emits HandlerEvent::MessageDropped, and
behaviour.rs counts it as failed_messages.non_priority += 1. So a
SlowPeer report also means our side did not flush in time, not only that
the remote handler is dead.

If the swarm task is starved for a while, or the uplink is saturated, every
peer gets reported in the same heartbeat. After six consecutive checks
disconnect_slow_peers then drops the whole peer set in one tick. Discovery
and the bootnode redial refill it, but the node eclipses itself for a while.

disconnect_graylisted_peers has the same shape, but graylisting needs
sustained invalid behaviour, so this new rule is the easier one to trip.

Two cheap ways to close it:

  • cap the number of peers one check may disconnect, or
  • skip the pass when more than some fraction of connected peers were
    reported, since that points at us rather than at them.

…hem per check

Gossipsub counts a forward that expired in a peer's send queue as the same
non-priority failure as a full queue. A saturated uplink or a stalled runtime
on our side therefore makes every peer look slow at once, and the check would
disconnect the whole peer set in one tick.

- A SlowPeer report now counts only when its priority or non-priority
  failures exceed SLOW_PEER_REPORT_THRESHOLD, the cut Lighthouse applies
  before penalizing. A dead stream fails every message queued for the peer;
  a few failures per heartbeat are more likely late forwards.
- One check disconnects at most MAX_SLOW_PEER_DISCONNECTS_PER_CHECK peers,
  the most failed first. The rest keep their streak and are logged as a
  warning, since many peers failing together points at our side.
@MegaRedHand

Copy link
Copy Markdown
Collaborator Author

@pablodeymo done

MegaRedHand added a commit that referenced this pull request Nov 5, 2026
): review fixes

fix(p2p): gate slow-peer disconnects on Lighthouse's threshold, cap them per check

Gossipsub counts a forward that expired in a peer's send queue as the same
non-priority failure as a full queue. A saturated uplink or a stalled runtime
on our side therefore makes every peer look slow at once, and the check would
disconnect the whole peer set in one tick.

- A SlowPeer report now counts only when its priority or non-priority
  failures exceed SLOW_PEER_REPORT_THRESHOLD, the cut Lighthouse applies
  before penalizing. A dead stream fails every message queued for the peer;
  a few failures per heartbeat are more likely late forwards.
- One check disconnects at most MAX_SLOW_PEER_DISCONNECTS_PER_CHECK peers,
  the most failed first. The rest keep their streak and are logged as a
  warning, since many peers failing together points at our side.

Updates nightly's #739 from fix/p2p-disconnect-slow-gossipsub-peers at
380fd29 to 8306770 (the branch head, 6b11692, adds only a merge of main).

Claude-Session: https://claude.ai/code/session_011vghNmndKPkZinHRkVTwVB
MegaRedHand added a commit that referenced this pull request Nov 5, 2026
…ient's blob pool (#769)

A fulu block whose custody columns have not all arrived waits on gossip and
on per-slot peer asks. The blob transactions it commits to usually passed
through the local execution client's pool first, so the node can often build
those columns itself, sooner. Fulu's p2p-interface asks honest nodes to do so
with engine_getBlobsV2, as Lighthouse and the other clients already do.

On a fulu block's first data-availability hold, with an execution client
configured, the chain actor spawns a task that asks engine_getBlobsV2 once
(1 s, the spec's timeout, outside the retry ladder: the call races gossip),
builds the custody columns still missing without verifying the KZG proofs
(the spec says to skip verification for local-EL blobs), and hands them to
p2p through the existing check_data_column_sidecars, now tagged with a
SidecarOrigin. For ExecutionLayer sidecars p2p runs no checks; it records
them in the seen cache, publishes them on their subnets, and forwards them
to the chain actor, which stores them and releases the block. Those two
steps are the spec's MUSTs for importing local-EL blobs, and p2p owns both.

V2 only: V3 answers partially, and a column needs a cell from every blob.
Gloas is not covered, since availability moves to the payload envelope.

On nightly, a Parked recheck keeps #765's ColumnArrival::Recheck, and the
CLAUDE.md peer-scoring line keeps #739's slow-peer text. Columns built from
the execution client write no block_events telemetry rows: #765's column
rows cover gossip-accepted and fetched columns only.

Cherry-picked from feat/engine-get-blobs-v2 at 39d5f07 (e0f69b3 plus a
merge of main).

Claude-Session: https://claude.ai/code/session_011vghNmndKPkZinHRkVTwVB
…lone

The per-check cap on slow-peer disconnects bounds the rate, not the
total: an uplink that stays too slow for gossip past the streak still
drains peers while discovery refills them. That case is deliberate:
bandwidth that slow is not something the node can solve from its side.
`disconnect_slow_peers` and `disconnect_graylisted_peers` each asked
gossipsub for our own peer score to tell beacon from lean, and only one
said why. `scoring_enabled` makes that one function, with the reason and
the test that pins it, so the two gates cannot drift apart.
The counts are only logged and ranked, and `usize` will not overflow on
real queue sizes, but a plain add panics in a debug build if it ever
does. Saturating keeps this bookkeeping off the panic path.
`disconnect_peer_id` fails only when the peer has no connection left, so
there is no pinned queue to free. Dropping its streak along with it is
right, since a peer that reconnects gets a fresh handler.
`several_reports_within_one_check_count_once_towards_the_streak` only
showed the reports did not trip the limit at once. It now carries the
streak to the limit, which fails if that check counted zero times or
more than once.

`a_returned_peer_starts_a_new_streak` never asserted the peer was
returned, so a `check` that returned nothing would have passed. It now
asserts the return, then a second full streak with only its own failures.
@MegaRedHand
MegaRedHand added this pull request to stack #798 November 17, 2026 22:12
@MegaRedHand

Copy link
Copy Markdown
Collaborator Author

/review

@github-actions

Copy link
Copy Markdown

🤖 Gemini Code Review

The implementation looks solid and well-tested.

  • Separating SlowPeers state tracking from the swarm makes the streak, threshold, and per-check cap logic easy to follow and unit-test in isolation.
  • Calling slow_peers.check() before the scoring_enabled(swarm) guard in disconnect_slow_peers (crates/net/p2p/src/swarm_adapter.rs:471-478) ensures the maps are still drained on Lean nodes and cannot grow unboundedly.
  • The failure threshold (SLOW_PEER_REPORT_THRESHOLD), streak requirement (SLOW_PEER_CHECK_LIMIT), and per-check cap (MAX_SLOW_PEER_DISCONNECTS_PER_CHECK) appropriately protect the node from dropping its peer set during transient local uplink/runtime stalls.

No issues found — LGTM.


Automated review by Gemini · gemini-4-argon · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Codex Code Review

  • [P2] Clear slow-peer history when the last connection closes — swarm_adapter.rs:299–304. Only SlowPeer events update the tracker. If a peer accumulates six qualifying checks’ reports, disconnects, and reconnects before the next timer tick, the stale reports can disconnect its fresh, healthy connection. Reset both reported and streaks on ConnectionClosed when num_established == 0, and add a reconnect regression test.

No consensus-processing code changes in this PR. Tests could not run because rustup attempted to write to a read-only directory.


Automated review by OpenAI Codex · gpt-6.1-sol · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Claude Code Review

Review: slow-peer disconnects in swarm_adapter.rs

The design is sound and well documented. The state machine in SlowPeers is simple and its memory is bounded: reported is cleared every check, and streaks keeps only peers reported in the last check. The tests cover the edge cases that matter: the threshold boundary, a streak reset, several reports within one check, and the deferred peers left over by the cap. I found no correctness bugs, only design points and nits.

Design and correctness

  1. The per-check cap does not detect a cause on our side (swarm_adapter.rs, MAX_SLOW_PEER_DISCONNECTS_PER_CHECK).

    • When deferred > 0, the code already knows "many peers are slow at once". The warning says this points at our own uplink or runtime, yet the node still disconnects 10 peers per check.
    • With a sustained saturated uplink, that drains the peer set at about 60 peers per minute. Discovery then refills it with peers who will look slow too. This churn adds handshake and discovery load to an uplink that is already saturated.
    • Consider a circuit breaker instead of, or in addition to, the rate cap. For example, if more than some fraction of connected peers (say 25–50%) hit the limit in one check, disconnect none of them and only warn.
    • A real dead stream is a per-connection fault, so it should almost never be correlated across many peers. A breaker would separate the two cases far better than a cap does. The docs say the case is "deliberately left alone", but the current behavior is not leaving it alone: it actively churns peers.
  2. Gating on scoring_enabled mixes two unrelated concerns (disconnect_slow_peers).

    • The memory pin comes from a dead handler, which is a gossipsub handler bug. It affects lean nodes exactly as much as beacon nodes.
    • Reusing "scoring is on" as a proxy for "this is beacon" works today, but it couples the two.
    • If lean is meant to be excluded (small devnets, few peers), consider an explicit flag. Then a later change to lean scoring will not silently enable or disable this.
    • At minimum, state in the doc comment why lean is excluded, not only that it is.
  3. The per-report threshold misses steady low-rate dead streams (record).

    • Gossipsub reports once per heartbeat, about every 700 ms, so roughly 14 reports per check. A dead peer failing 5–10 messages per heartbeat (60–140 per check) never counts toward a streak.
    • The docs acknowledge this for pruned or low-traffic peers.
    • An alternative is to sum all reports between checks and apply a per-check threshold. That still filters out sporadic expiries but catches a steady trickle.
    • Following Lighthouse's cut is a reasonable default, so this is optional.
  4. A peer can be double-counted across the two disconnect paths.

    • disconnect_graylisted_peers runs immediately before disconnect_slow_peers.
    • A peer that is both graylisted and slow can get disconnect_peer_id → Ok twice in the same tick, because its connections are still closing. That increments both counters and logs twice.
    • This is minor. You could collect the graylisted set and skip those peers in the slow pass, or accept the overlap and note it in metrics.md.

Nits

  • SLOW_PEER_CHECK_LIMIT: u32 forces usize::try_from(...).unwrap() in the tests (checks_in_a_streak). Making the constant usize removes the conversion. streaks could then store (usize, usize).
  • streaks: HashMap<PeerId, (u32, usize)>: a small named struct (Streak { checks, failed }) would read better than tuple destructuring in check().
  • sort_unstable_by(|(_, a), (_, b)| b.cmp(a)) can be written as sort_unstable_by_key(|&(_, f)| Reverse(f)). Ties are broken arbitrarily, which is fine but could be mentioned.
  • With MissedTickBehavior::Skip, a runtime stall merges several intervals' reports into one check. That is the right outcome (a stall counts once rather than many times), but a one-line comment would stop someone "fixing" it to Burst.
  • The CLAUDE.md bullet is getting very long. Consider moving the slow-peer details to docs/beacon_wire.md only and linking to it.

Tests

The unit tests are thorough for SlowPeers. Nothing tests the gating in disconnect_slow_peers, or that a SlowPeer event reaches record. That is acceptable given that swarm-level tests are expensive.

Overall the change looks good to merge. Item 1 is the one I'd most like to see addressed or explicitly discussed.


Automated review by Claude · claude-opus-5-5 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 DeepSeek Code Review

This PR is well-scoped to p2p peer management; the consensus/fork-choice/state-transition code is untouched. The SlowPeers streak/threshold/cap logic is clear and the tests cover the important edge cases.

One issue that likely blocks compilation:

  1. Tracing macro field syntax — crates/net/p2p/src/swarm_adapter.rs, in the new disconnect_slow_peers block:

    • warn!(disconnecting, deferred, "...")
    • info!(%peer_id, failed, "...")

    disconnecting, deferred, and failed are bare identifiers. tracing macros do not support struct-style field shorthand; fields must be recorded as name = value, ?name, or %name.

    Suggested changes:

    warn!(
        disconnecting = disconnecting,
        deferred = deferred,
        "More peers kept failing to take gossipsub messages than one check disconnects, \
         which points at our own uplink or runtime; the rest wait for a later check"
    );

    and:

    info!(
        %peer_id,
        failed = failed,
        "Disconnecting peer that kept failing to take gossipsub messages"
    );

    or use %failed if display formatting is preferred.

  2. Minor: in the tests, checks_in_a_streak() uses usize::try_from(SLOW_PEER_CHECK_LIMIT).unwrap(); SLOW_PEER_CHECK_LIMIT as usize would be simpler, but this is not a correctness issue.

Otherwise, the disconnection rate limiting, streak reset behavior, and threshold handling look correct and appropriately defensive against self-inflicted mass disconnects.


Automated review by DeepSeek · deepseek-v4-pro · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Kimi Code Review

No blocking correctness, security, or performance issues found.

  • crates/net/p2p/src/swarm_adapter.rs: The streak handling is coherent: reports are consolidated per check, missed or sub-threshold checks reset streaks, disconnected peers restart tracking, and deferred peers correctly retain their streaks.
  • The per-check disconnect cap and descending failure-count ordering provide reasonable protection against local uplink/runtime problems causing simultaneous mass disconnects.
  • Saturating additions avoid counter overflow for failure totals, and metrics increment only when disconnect_peer_id succeeds.
  • The beacon/lean gate is shared consistently with graylist disconnects.
  • Unit tests cover the important state transitions, including threshold boundaries, multiple reports per check, cap ordering, and deferred peers.

The remaining limitations—low-traffic dead streams staying below the threshold and sustained local saturation still causing capped peer churn—are explicitly documented and appear intentional.


Automated review by Kimi (Moonshot AI) · kimi-k3.1-pro · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Grok Code Review

The slow-peer tracking is careful and the streak, threshold, and cap tests match the stated rules. One behavior still works against the failure mode this is trying to contain.

  1. In disconnect_slow_peers, a full cap is treated as evidence the fault is local, then the busiest peers are disconnected anyway. SlowPeers::check already drops those peers' streaks before that decision, so the caller cannot skip the disconnect and keep them as candidates. When deferred > 0, return without disconnecting, and only clear streaks for peers whose connections actually close. As written, a saturated uplink still sheds up to 10 peers every 10 s (about 60/minute) while discovery refills them, which adds handshake load on top of the stall. Ranking by cumulative failed makes that worse: the peers we enqueue the most for (column traffic) fail the most and are cut first, whether the handler is dead or our send path is slow.

  2. SlowPeer is recorded and then forwarded on event_tx. Confirm the actor's gossipsub match ignores it. A wildcard warn on an unhandled variant would log every heartbeat for every slow peer.

Residual low-traffic dead streams staying under SLOW_PEER_REPORT_THRESHOLD is an acceptable, documented gap. The lean gate via scoring_enabled matches the graylist pass.


Automated review by Grok (xAI) · grok-4.7 · custom prompt

@pablodeymo
pablodeymo added this pull request to the merge queue Nov 18, 2026
Merged via the queue into main with commit 47542ea Nov 18, 2026
14 checks passed
@pablodeymo
pablodeymo deleted the fix/p2p-disconnect-slow-gossipsub-peers branch November 18, 2026 20:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants